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DETAILED ACTION 

Claim Objections 

1 . Claims 1 0-1 2 are objected to under 37 C.F.R 1 .75(d) because of the 
following informalities: the term "processed video stream" lacks antecedent basis. 
For the purposes of examination the phrase will be interpreted as "processed 
video sequence". Appropriate correction is required. 

Claim Rejections - 35 USC § 102 

1 . The following is a quotation of the appropriate paragraphs of 35 
U.S.C. 102 that form the basis for the rejections under this section made in this 
Office action: 

A person shall be entitled to a patent unless - 

(a) the invention was known or used by others in this country, or patented or described in a printed 
publication in this or a foreign country, before the invention thereof by the applicant for a patent. 

2. Claims 1-5, 7-13, 15-18 and 23-25 are rejected under 35 U.S.C. 102(a) as 
being anticipated by Caspi et al. ("Spatio-Temporal Alignment of Sequences", Y. 
Caspi and M. Irani, IEEE Trans, on Pattern Analysis and Machine Intelligence, 
Vol. 24 No. 11, pp. 1409-1424, November 2002). 

Regarding claim 1 , Caspi discloses a method for identifying 
correspondence between an original video sequence comprising a plurality of 
original frames and a processed video sequence comprising a plurality of 
processed frames (Caspi's system "establishes correspondences both in time 
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and in space between multiple sequences"; Section 1 Page 1 ), the method 
comprising: 

dividing the processed video sequence into a plurality of processed sets, 
each processed set having one or more processed frames (Caspi evaluates 
individual space-time points in each sequence, evaluating each frame of the 
second sequence individually; Section 2 Page 3); 

identifying, for each processed set, one or more original sets from the 
original video sequence, wherein each original set comprises one or more 
original frames (Caspi's spatio-temporal misalignment P is defined between two 
space-time points, x and x', each point necessarily belonging to a single frame 
in the reference sequence S and the second sequence S' t respectively; Section 2 
Page3); 

and two or more original sets are identified for at least one processed set 
(In the minimization of error function (4) a single space-time point x and 
associated frame in S is compared to all space-time points x' and associated 
frames in S'; Section 3.2 Page 8); 

' generating a mapping for each original set corresponding to each 
processed set, wherein the mapping defines, for the original set, a mapped set 
that approximates the corresponding processed set (the parametric model P 
defines the misalignment between S and S'; Section 2 Page 3); and 

the mapping minimizes a local prediction error between the mapped set 
and the corresponding processed set; and selecting, for each processed set, the 
original set whose mapping minimizes an accumulated prediction error for the 
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processed video sequence (Equations (4) and (5) minimize prediction error P 
between sequences, with P consisting individual displacements u between 
frames; Section 3.2 Page 8). 

Regarding claim 2, Caspi further discloses the method as detailed above 
wherein each original set consists of one or two original frames; each mapped 
set consists of a single mapped frame; and each processed set consists of a 
single processed frame (Caspi relates a single frame in the reference sequence 
to one in the second sequence via space-time points; Section 2 Page 3). 

Regarding claim 3, Caspi further discloses the method as detailed above 
wherein generating each mapping involves temporal registration (Caspi 
calculates a "spatio-temporal displacement u", comprising a temporal portion 
used to align the two sequences; Section 2 Page 3). 

Regarding claim 4, Caspi further discloses the method as detailed above 
wherein generating each mapping involves at least one of spatial registration and 
histogram registration (Caspi calculates a "spatio-temporal displacement u", 
comprising a spatial portion used to align the two sequences; Section 2 Page 3). 

Regarding claim 5, Caspi teaches the invention of claim 4, wherein 
generating each mapping involves spatial registration (Caspi calculates a "spatio- 
temporal displacement u", comprising a spatial portion used to align the two 
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sequences; Section 2 Page 3) and histogram registration (Caspi recovers "spatial 
and temporal alignment parameters between the two sequences directly from 
sequence brightness variations", where the histogram is a representation of 
image brightness; Section 3.2 Page 8). 

Regarding claim 7, Caspi teaches the invention claimed in 3 wherein the 
temporal registration involves minimizing the local prediction error between a 
weighted sum of two original frames and a corresponding processed frame 
("Sequence-to-sequence alignment, on the other hand is not restricted to 
physical ("integer") image frames."; Section 4.1 (ii) Page 15). 

Regarding claim 8, Caspi teaches the invention of claim 1, wherein one or 
more constraints are applied to limit the number of different original sets that are 
identified for each processed set (the "outlier rejection mechanism", Section 3.1 
Page 7 Paragraph 2, in step 5 of the algorithm detailed in Section 3.1 Page 5). 

Regarding claim 9, Caspi teaches the invention of claim 8, wherein at 
least one constraint is a causal constraint that specifies that no original frames 
displayed in the past can be used to generate a current processed frame (The 
feature based alignment algorithm detailed in section 3.1 utilizes "feature 
trajectories", which can "uniquely define: (i) the spatial transform, (ii) the temporal 
transform". All trajectories of real objects obey causality, and the "outlier 
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rejection mechanism" of step 5 would remove any non-causal hypothesis from 
consideration; Section 3.1 Page 5). 

Regarding claim 10, Caspi teaches the invention of claim 1 wherein a first 
selected original set corresponds to the first processed frame in the processed 
video sequence; and each other selected original set for each other processed 
frame depends on a selected original set corresponding to a previous processed 
frame (The interpretation of "first processed frame" will be taken as the first frame 
in a chosen subsequence of the processed set, not necessarily the first frame of 
the entire processed set. Under this interpretation, the feature based sequence 
alignment method of 3.1 using feature trajectories aligns two video sequences 
via causal trajectory information. The alignment is based on causal information, 
such that all points chosen after a given point depend on it, i.e. when 
constructing feature trajectories the method will "detect and track feature points" 
where tracking indicates some predictive aspect of following a point, Section 3.1 
Page 5). 

Regarding claim 11, Caspi teaches the invention of claim 10, wherein the 
dependence between selected original sets is based on one or more constraints 
(the "threshold" of step (5) of the algorithm, Section 3.1 Page 5). 

Regarding claim 12, Caspi teaches the invention of claim 1 1 , wherein at 
least one constraint is a causal constraint that specifies that no original frames 



Application/Control Number: 10/792,073 Page 7 

Art Unit: 2609 

displayed in the past can be used to generate a current processed frame (Using 
the same grounds for rejection as claim 9, the feature based alignment algorithm 
detailed in section 3.1 utilizes "feature trajectories", which can "uniquely define: 
(i) the spatial transform, (ii) the temporal transform". All trajectories of real 
objects obey causality, and the "outlier rejection mechanism" of step 5 would 
remove any non-causal hypothesis from consideration; Section 3.1 Page 5). 

Regarding claim 13, Caspi teaches the method of claim 1 wherein at least 
one original frame is not included in any mapping (Temporal misalignment 
results when the two input sequences have a time-shift (offset) between them", 
where any video portion not common to both sequences would not be included in 
any mapping; Section 2 Page 3). 

Regarding claim 16, Caspi teaches the invention of claim 1 , wherein the 
local prediction error between each mapped set and the corresponding 
processed set is a function of a matching term and a context term (The algorithm 
laid out in steps (1)-(8) in Section 3.1 Page 5-6 identifies both elements shown 
below); 

the matching term characterizes differences between the mapped set and 
the corresponding processed set (Equation (1) calculates the displacement error 
function between both sets and is used in (7) of the algorithm, Section 3.1 Page 
6); and 
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the context term corresponds to a cost associated with one or more 
contextual constraints applied to a temporal relationship between the original and 
processed video sequences ("a single pair of corresponding trajectories can 
uniquely define ... the temporal transformation" Caspi Section 3.1 Page 5). 

Regarding claim 17, Caspi teaches the invention of claim 16, wherein the 
matching term corresponds to a mean squared error between the mapped set 
and the corresponding processed set (Equation (1) calculates the displacement 
error function between both sets using the sum of squared errors, Section 3.1 
Page 6). 

Regarding claim 18, Caspi teaches the invention of claim 16, wherein the 
context term enforces a causal constraint that no previously displayed frames 
can be subsequently processed (The "feature trajectories" of real objects obey 
causality; Section 3.1 Page 5). 

Regarding claim 23, Caspi teaches the invention claimed in 1 further 
comprising the step of adjusting the processed video sequence to correct for one 
or more misalignments between the original and processed video sequences 
(Caspi's system aligns two sequences in Figure 4 Page 10). 

Regarding claim 24, Caspi teaches the invention of claim 23, wherein the 
one or more misalignments include at least one of spatial misalignment and 
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histogram misalignment (Caspi's system aligns two sequences in the spatial 
domain in Figure 4 Page 10). 

Regarding claim 25, Caspi teaches claim 24 wherein the one or more 
misalignments include both spatial misalignment (Caspi's system aligns two 
sequences in the spatial domain in Figure 4 Page 10) and histogram 
misalignment (And uses image brightness (histogram) matching in Figure 8 Page 
14). 

Claim Rejections - 35 USC § 103 

3. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for 
all obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described 
as set forth in section 102 of this title, if the differences between the subject matter sought to 
be patented and the prior art are such that the subject matter as a whole would have been 
obvious at the time the invention was made to a person having ordinary skill in the art to which 
said subject matter pertains. Patentability shall not be negatived by the manner in which the 
invention was made. 

4. Claim 14 is rejected under 35 U.S.C. 103(a) as being unpatentable over 
the combination of Caspi et al. ("Spatio-Temporal Alignment of Sequences", Y. 
Caspi and M. Irani, IEEE Trans, on Pattern Analysis and Machine Intelligence, 
Vol. 24 No. 11, pp. 1409-1424, November 2002) and Claus et al. (US 4,786,979). 

Regarding claim 14, Caspi teaches the method of claim 1. 
Caspi does not teach the method wherein one processed frame is a 
repetition of the immediately preceding processed frame. 
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Claus teaches an exemplary system of video processing that introduces a 
sequentially duplicate frame (Claus Fig. 1). 

It would have been obvious at the time of the invention to one of ordinary 
skill in the art to use the processing method of Claus as the means to create the 
processed sequence in Caspi because the method taught by Claus is pervasive 
in the art of video processing ("Currently, the most popular method of converting 
24 frames per second (fps) film to 60 fields/s video is to repeat each odd- 
numbered frame for 3 fields and each even-numbered frame for 2 fields", 
Introduction Hilman et al. ("Using Motion-Compensated Frame-Rate Conversion 
for the Correction of 3:2 Pulldown Artifacts in Video Sequences", Hilman et ah, 
IEEE Trans. On circuits and systems for video technology, Vol. 10, No.6, pp. 
869-877, September 2000)) 

5. Claim 21 rejected under 35 U.S.C. 103(a) as being unpatentable over the 
combination of Delannay et al. ("Compensation of Geometrical Deformations for 
Watermark Extraction in the Digital Cinema Application", Delannay et al., SPIE 
Security and Watermarking of Multimedia Contents III, Vol. 4314, pp. 149-157, 
August 2001 ) and Caspi et al. ("Spatio-Temporal Alignment of Sequences", Y. 
Caspi and M. Irani, IEEE Trans, on Pattern Analysis and Machine Intelligence, 
Vol. 24 No. 1 1 , pp. 1409-1424, November 2002). 

Regarding claim 21, Delannay teaches the method of processing a video 
sequence, wherein the video sequence has been generated by capturing with a 
camcorder a display of the original video sequence ("a spectator filming the 
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projected movie with a handy cam at the back of the theatre." Delannay Section 
2.1 Page 150). 

Delannay does not teach the method of claim 1 . 

Caspi teaches all elements of the method of claim 1 . 

It would have been obvious at the time the invention was made to one of 
ordinary skill in the art to provide the method of Delannay with the spatio- 
temporal alignment of Caspi to ensure proper watermark extraction. Delannay 
uses a watermark extraction technique that is done "with the help of the 
unmodified content" (Delannay Section 3 Page 153). Furthermore, Delannay 
tries to "obtain a image very close to the original" (Delannay Section 3 Page 1 53) 
in order to recover the watermark features because the system relies on 
matching features between the original and unprocessed video sequences. 
Caspi's system is an extension of that same principle, and could be employed in 
Delaney's system to obtain an image sequence closer to the original to properly 
recover a watermark from a video sequence. 

6. Claim 22 is rejected under 35 U.S.C. 103(a) as being unpatentable over 
the combination of Schumann et al. (US 6,285,774 B1 ) and Caspi et al. ("Spatio- 
Temporal Alignment of Sequences", Y. Caspi and M. Irani, IEEE Trans, on 
Pattern Analysis and Machine Intelligence, Vol. 24 No. 1 1, pp. 1409-1424, 
November 2002). 

Regarding claim 22, Schumann teaches identifying a watermark in an 
original frame; and determining whether a corresponding processed frame has a 
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similar watermark (Mark extraction system 122 Fig. 1 , where the processed 
frame is derived from the unauthorized copy 138 Fig.1). 

Schumann does not teach the method of claim 1 . 

Caspi teaches all elements of claim 1 , as detailed above. 

It would have been obvious at the time the invention was made to one of 
ordinary skill in the art to provide the system of Schumann with the spatio- 
temporal registration method of Caspi to overcome any misalignment between 
the unauthorized copy and the original copy of the movie to improve accuracy of 
the mark extraction system. Delannay et al. ("Compensation of Geometrical 
Deformations for Watermark Extraction in the Digital Cinema Application", 
Delannay et al., SPIE Security and Watermarking of Multimedia Contents III, Vol. 
4314, pp. 149-157, August 2001) tries to "obtain a image very close to the 
original" (Delannay Section 3 Page 153) in order to recover watermark features. 
Caspi's system is an extension of that same principle, and could be used to 
match a reference sequence to a processed sequence in Schumann's invention 
to recover a watermark. 

Claims 26 and 27 are rejected under 35 U.S.C. 203(a) as being 
unpatentable over the combination of Caspi et al. ("Spatio-Temporal Alignment of 
Sequences", Y. Caspi and M. Irani, IEEE Trans, on Pattern Analysis and 
Machine Intelligence, Vol. 24 No. 11, pp. 1409-1424, November 2002) and Caspi 
et al. (US 2002/0094135 A1). 

Regarding claim 26, Caspi IEEE teaches the method of claim 1. 
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Caspi IEEE does not teach a machine-readable medium, having encoded 
thereon program code, wherein, when the program code is executed by a 
machine, the machine implements the method of claim 1. 

Caspi '135 teaches a "preferred software embodiment" (Caspi '135 
Paragraph [0176]) to implement a video alignment algorithm. 

It would have been obvious at the time of invention to one of ordinary skill 
in the art to provide the method of Caspi IEEE with the software embodiment of 
Caspi '135 in order to execute the method on digital data using a general- 
purpose computer. 

Regarding claim 27, Caspi IEEE teaches the method of claim 1 . 
Caspi IEEE does not teach an apparatus with means for executing the 
method of claim 1 . 

Caspi '135 teaches a "Silicon Graphics computer" (Caspi '135 Paragraph 
[0176]) being configured to execute a method of video alignment. 

It would have been obvious at the time of invention to one of ordinary skill 
in the art to provide the method of Caspi IEEE with the apparatus embodiment of 
Caspi '135 to have a way to execute the video registration method and to provide 
user interaction to the method. 



r 
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Allowable Subject Matter 

7. Claims 6, 1 5, 19, and 20 objected to as being dependent upon a rejected 
base claim, but would be allowable if rewritten in independent form including all 
of the limitations of the base claim and any intervening claims. 

Regarding claim 6, the most pertinent prior art is Caspi et al. as cited 
above. Although Caspi does teach each of temporal, spatial, and histogram 
registration independently; Caspi does not teach a method of using all three in 
the same embodiment and iteratively fixing two while optimizing the third. 

Regarding claim 15, the most pertinent prior art is Caspi et al. as cited 
above. Caspi teaches a causal constraint placed on the matching function, 
however this constraint is not a monotonically increasing function nor does it 
increase from a selected frame with the largest index. 

Regarding claim 19, the most pertinent prior art is Caspi et al. While 
Caspi teaches a local prediction error comprising two terms, Caspi does not 
teach penalizing repetition and dropping of original frames. 

Regarding claim 20, the most pertinent prior art is again Caspi et al. Caspi 
teaches a causal constraint placed on the matching function, however this 
constraint is not a monotonically increasing function nor does it increase from a 
selected frame with the largest index. 



Conclusion 

8. The prior art made of record and not relied upon is considered pertinent to 
applicants disclosure. Hu (US 6,259,477 B1) teaches a system for aligning video 
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sequences in the spatial and temporal domains for the purposes of fidelity 
comparison between a reference and processed sequence. Hu uses a less 
robust method of spatio-temporal alignment than applicant that relies exclusively 
on scene transitions. Similarly, Overton (US 5,894,324) teaches the method of 
aligning two video sequences using partial picture, again an approach not nearly 
as robust as applicant's own. Joly (US 7,043,684 B2) discloses a method for 
synchronization of arbitrary data streams with the same content, with one being 
processed. Joly discloses a robust temporal alignment method but does not 
teach a complete spatial alignment method. Lastly, Caspi ("Alignment of non- 
overlapping sequences", Caspi et al., IEEE Eighth International conference on 
Computer Vision, Vol. 2, pp. 76-83, July 2001) is an earlier version of the Caspi 
paper referenced above, and could have been used in replacement of the later 
version for most purposes. 

Any inquiry concerning this communication or earlier communications from 
the examiner should be directed to Jared W. Radkiewicz whose telephone 
number is (571) 270-1577. The examiner can normally be reached on 8:00 - 
5:00 EST. 

If attempts to reach the examiner by telephone are unsuccessful, the 
examiner's supervisor, Brian P. Werner can be reached on (571 ) 272-7401 . The 
fax phone number for the organization where this application or proceeding is 
assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from 
the Patent Application Information Retrieval (PAIR) system. Status information 
for published applications may be obtained from either Private PAIR or Public 
PAIR. Status information for unpublished applications is available through 
Private PAIR only. For more information about the PAIR system, see http://pair- 
direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll- 
free). If you would like assistance from a USPTO Customer Service 
Representative or access to the automated information system, call 800-786- 
9199 (IN USA OR CANADA) or 571-272-1000. 
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